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Foreword 


This  report  was  developed  by  Shantu  Dhar  and  Robert  S.  Matthews  of  the  Industrial  Technology 
Institute  (ITI)  under  a National  Institute  of  Standards  and  Technology  (NIST)  task  order  contract 
number  43NANB020484.  ITI  is  a not-for-profit  research  and  development  organization  located 
in  Michigan,  whose  mission  is  to  advance  the  competitiveness  of  American  manufacturing 
organizations.  ITI  technical  staff  is  experienced  and  educated  in  manufacturing-related 
technologies  and  organizational,  economic,  and  business  issues.  One  area  (of  many)  where  ITI 
and  NIST  interests  coincide  is  in  the  development  and  conformance  testing  of  ST^P.  NIST 
published  the  National  PDES  Testbed  Development  Plan  for  a STEP  Conformance  Testing 
Service  (NISTIR  4641),  which  defines  an  important  need  for  the  conformance  testing  (CT)  of 
STEP  products. 

The  work  described  in  this  document  was  funded  by  the  U.S.  Government’s  Department  of 
Defense  Computer-aided  Acquisition  and  Logistic  Support  (CALS)  initiative.  Although  the 
CALS  Evaluation  and  Integration  Office  and  NIST  do  not  necessarily  endorse  the 
recommendations  found  here,  it  is  important  to  make  public  such  recommendations  for 
discussion.  This  report  focuses  on  important  issues  that  will  arise  during  an  effort  to  offer  a 
full-scale  STEP  conformance  testing  service  to  U.S.  industry.  The  content  presented  here  draws 
from  ITI’s  past  conformance  testing  experience  in  order  to  provide  a perspective  of  the  real-life 
problems  associated  with  developing  and  offering  testing  services.  Thus,  to  the  decision-maker 
and  funding  agency  considering  active  participation  in  STEP  CT,  this  rej^rt  identifies  major 
issues  confronting  CT  in  gener^,  and  STEP  CT  in  the  United  States  in  particular.  It  also  offers 
insight  into  the  direction  that  U.S.  activity  should  proceed. 

This  report  is  not  subject  to  copyright. 
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Executive  Summary 


The  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  is  an  emerging  set  of  standards 
that  will  revolutionize  the  use  and  exchange  of  product  information  by  manufacturing 
organizations.  In  order  for  STEP  to  achieve  this  objective  of  free  exchange  and  use  of  product 
data,  STEP-conformant  products  based  on  standardized  application  protocols  have  to  be 
available  to  users.  Moreover,  since  such  products  will  interact  and  transmit  information  among 
each  other,  it  is  imperative  that  they  "interoperate"  (work  together  or  share  data  appropriately). 

The  National  Institute  of  Standards  and  Technology  (NIST)  actively  participates  in  the  STEP 
development  effort  to  support  U.S.  industry.  NIST  and  other  U.S.  activities  contributing  to 
STEP  are  known  as  PD^  (Product  Data  Exchange  using  STEP).  NIST  is  developing  a 
nation-wide  Product  Data  Exchange  Network,  ‘ along  with  associated  long-range  strategic 
development  plans.  The  objective  of  this  activity  is  to  enable  the  broad-based  adoption  of 
STEP-conformant  products. 

The  objective  of  this  report  is  to  evaluate  the  need  for  a conformance  testing  (CT)  service  for 
STEP  implementations,  and  identify  the  steps  that  should  be  taken  to  develop  an  appropriate  CT 
service.  Two  important  assumptions  underlie  the  conclusions  presented  in  this  report.  First, 
experiences  gain^  from  CT  services  of  other  standards  implementations  can  be  adapted  to 
STEP.  Second,  STEP  is  not  a single  standard  but  a collection  of  standards.  Among  this 
collection  are  application  protocol  (AP)  standards,  on  which  conformance  testing  of 
implementations  will  be  based.  An  application  protocol  defines  the  context  for  the  use  of 
product  data  and  specifies  the  use  of  the  standard  in  that  context  to  satisfy  an  industrial  need. 


This  report  is  deliberately  freestanding  and  fairly  generic.  Many  of  the  issues  raised  here  have 
been  or  are  being  considered  by  ISO  TC184/SC4,^the  international  organization  responsible  for 
STEP  standardization. 


Conformance  testing  for  standards  such  as  STEP  is  a means  of  increasing  the  likelihood  that 
two  different  systems  operating  under  the  same  standard  will  work  together.  It  is  based  on  the 
concept  that  evaluating  implementations  of  an  AP  against  the  standard  itself,  then  giving  the 
vendor  time  to  make  any  corrections,  will  eliminate  most  obstacles  to  successful 
interoperability. 

When  implementations  of  standards  for  information  representation  and  exchange  were  few  in 
number,  the  evaluation  of  products  required  only  the  straightforward  operation  of  determining 
whether  they  interoperated.  Testing  every  product  with  every  other  through  interoperability 
explodes  to  a complex  task  as  more  products  exist.  This  is  why  conformance  testing  (CT)  is 
the  most  important  method  of  evaluating  a product’s  adherence  to  a standard.  Conformance 
testing  does  not  remove  the  need  for  interoperability  testing,  but  it  can  dramatically  reduce  the 
amount  of  effort  required  to  get  two  differing  systems  to  work  together.  The  cost  of  CT  is  not 
insignificant,  but  if  execut^  properly,  it  reduces  the  overall  cost  of  getting  systems  to 
interoperate. 


‘Simon  Frechette  and  Kevin  Jurrens,  Product  Data  Exchange  Network.  NISTIR  4431,  September  1990. 

^International  Organization  for  Standardization  (ISO)  Technical  Committee  on  Industrial  Automation 
(TC184)  Sub-Committee  on  Industrial  Data  and  Global  Manufacturing  Programming  Languages  (SC4). 
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The  four  main  participants  concerned  with  CT  are  the  standard  bodies  that  develop  the 
standards,  the  vendors  who  build  standard-conformant  products,  the  users  who  require  these 
products  and  need  the  assurance  the  products  will  interoperate,  and  the  testing  community 
which  includes:  testing  laboratories,  accreditation  authorities,  and  certification  b^ies.  To  be 
successful,  CT  needs  to  satisfy  the  needs  of  all  these  entities.  It  must  reflect  the  true  intent  of 
the  standard  in  question,  be  inexpensive  in  the  long  run,  and  build  the  users’  confidence  in  the 
value  CT  provides. 

There  are  some  strategic  issues  to  consider  when  planning  for  STEP  CT.  The  involvement  of 
U.S.  bodies,  particularly  the  Department  of  Defense  (DoD),  is  critical.  Since  DoD  and  its 
suppliers  will  be  major  users  of  STEP,  the  indigenous  development  of  CT  capabilities  is 
essential.  Another  strategic  issue  is  that  of  CT  system  openness.  Once  developed,  STEP  CT 
systems  should  be  publicly  available  to  vendors  and  users  to  perform  in-house  testing  to  help 
in  product  and  application  development. 

A large  number  of  procedural  and  administrative  issues  are  associated  with  offering  STEP  CT. 
To  begin  with,  a single  or  small  number  of  STEP  APs  should  be  used  to  evaluate  CT  services. 
Then  the  experience  gained  in  this  preliminary  exercise  should  be  applied  toward  additional 
APs.  Since  funding  is  an  important  factor,  major  beneficiaries  of  these  services  would  be 
expected  to  fund  a CT  development  activity. 

A number  of  recommendations  are  made  as  a result  of  this  study.  Among  the  most  important 
are: 

• The  United  States  should  create  its  own  STEP  CT  system  and  not  rely  on  other 
countries. 

• The  CT  system  itself  should  be  made  widely  available,  and  in-house  testing  should 
be  encouraged  prior  to  formally-supervised  CT. 

• CT  system  development  should  start  small,  both  in  number  of  application  protocols 
addressed,  and  in  the  facility  established  (resources  and  equipment)  to  undertake 
development. 

• CT  systems  should  be  made  highly  automated  for  low-cost  operation  and  high 
accuracy. 
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i.  Introduction 


The  Standard  for  the  Exchange  of  Product  Model  Data  (STEP)  is  an  emerging  set  of  standards 
that  will  revolutionize  the  use  and  exchange  of  product  information  by  manufacturing 
organizations.  In  order  for  STEP  to  achieve  this  objective  of  free  exchange  and  use  of  product 
data,  STEP-conformant  products  based  on  standardized  application  protocols  have  to  be 
available  to  users.  Moreover,  since  such  products  will  interact  and  transmit  information 
between  each  other,  it  is  imperative  that  they  "interoperate"  (work  together  or  share  data 
appropriately). 

Conformance  testing  is  the  evaluation  of  a product  to  see  whether  it  meets  a particular  standard. 
Conformance  testing  (CT)  for  the  products  based  on  technologies  like  STEP  (Standard  for  the 
Exchange  of  Product  Model  Data),  where  many  suppliers  and  products  must  interoperate,  is 
critical  to  STEP’S  rapid  adoption  by  implementors  and  users.  CT  increases  the  user’s 
confidence  that  the  products  will  work.  TTie  initial  planning,  organizational  interfacing,  and 
prototyping  of  a CT  system  will  be  most  effective  if  coordinated  with  STEP  standards 
development  activities.  These  activities  include  those  associated  with  application  protocol 
prototyping  and  validation,  STEP  standardization  activities,  and  early  commercial  product 
development.  In  particular,  AP  validation  activities  can  contribute  test  data  and  software  to  the 
development  of  CT. 

The  objective  of  this  report  is  to  evaluate  the  need  for  a conformance  testing  (CT)  service  for 
STEP  implementations,  and  identify  the  steps  that  should  be  taken  to  develop  an  appropriate  CT 
service.  Two  important  assumptions  underlie  the  conclusions  presented  in  this  report.  First, 
experiences  gain^  from  CT  services  of  other  standards’  implementations  can  be  adapted  to 
STEP.  Second,  STEP  is  not  a single  standard  but  a collection  of  standards.  Among  this 
collection  are  application  protocol  (AP)  standards,  on  which  conformance  testing  of 
implementations  will  be  based.  An  application  protocol  defines  the  context  for  the  use  of 
product  data  and  specifies  the  use  of  the  standard  in  that  context  to  satisfy  an  industrial  need. 

The  following  philosophy  should  guide  CT  development  efforts.  Foremost,  CT  and  associated 
tools  must  help,  not  hinder,  market  adoption  of  conformance-tested  products.  CT  development 
efforts  can  lose  sight  of  this  objective  if  the  developers  are  not  careful.  Tools  or  services 
which  perform  to  CT  specifications  could  still  be  technically  cumbersome,  too  expensive,  or 
politicdly  unacceptable  because  of  a lack  of  consensus.  To  overcome  such  pitfalls,  it  is  key 
for  CT  developers  to  regularly  check  decisions  from  the  perspective  of  how  will  this  help 
market  adoption  and  stimulate  fhe  availability  of  useful  products.  By  bearing  this  in  mind,  CT 
will  stimulate  increasing  buyer  confidence  in  STEP  standards  and  STEP-based  products. 


1,1  Organization  of  this  Report 

In  addition  to  this  introduction,  this  report  has  four  major  sections.  Recommendations  on  a 
specific  issue  are  made,  as  appropriate,  in  the  subsection  dealing  with  that  issue. 

• Section  2 identifies  CT  issues  and  makes  recommendations  for  dealing  with  such 
issues  for  STEP. 
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• Section  3 discusses  evaluation  procedures  which  include  issues  relating  to  the 
abstract  and  executable  test  suites  (ATSs  and  ETSs)  and  tools,  and  evaluation  of  CT 
systems  and  testing  laboratories. 

• Section  4 proposes  a set  of  milestones  for  the  realization  of  STEP  CT  services  in 
the  United  States. 

• Section  5 concludes  the  report  with  a summary  of  the  recommendations. 


1.2  Definitions  and  References 

Those  terms  most  frequently  used  in  this  report  are  defined  in  Section  6.  Unless  otherwise 
noted,  the  definitions  are  extracted  from  ISO  CD  10303-31,  "Conformance  Testing 
Methodology  and  Framework,  General  Concepts,"  of  15  November  1990. 

In  addition  to  ISO  CD  10303-31,  the  following  documents  have  been  used  in  the  preparation 
of  this  report: 

ISO  DIS  9646,  OSI  Conformance  Testing  Methodology  and  Framework,  Parts  1,2,  4, 
and  5.  [Available  through  the  ISO  Secretariat:  1,  rue  de  Varembe,  Case  postale  56, 
CH-1211  Geneva  20,  Switzerland.] 

ISO/IEC  Guide  25,  General  Requirements  for  the  Technical  Competence  of  Testing 
Laboratories.  [Available  through  the  ISO  Secretariat:  1,  rue  de  Varembe,  Case  postale 
56,  CH-1211  Geneva  20,  Switzerland.] 

ISO  WD  10303-1,  "STEP  Overview  and  Fundamental  Principles,"  August  1990. 
[Available  through  ISO  TC184/SC4  Secretariat:  NIST,  220,  A127,  Gaithersburg,  MD 
20899,  USA.] 

m Technical  Report  87-14.1,  Test  Coverage  Analysis  and  Measurement  (TCAM)  - A 
Practical  Approach  to  Determining  Coverage.  [Available  through  ITI:  P.O.  Box  1485, 
Ann  Arbor,  MI  48106,  USA.] 
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2,  Conformance  Testing  Issues  and  Recommendations 


This  section  discusses  various  issues  relevant  to  STEP  CT.  The  approach  and  structure  of  this 
report  have  been  determined  through  discussions  between  NIST’s  contract  representatives  and 
rn  about  the  issues  that  need  to  be  addressed.  These  discussions  have  led  to  the  following 
major  areas  of  concern: 

• High-Level  Issues 

• Strategic  Issues 

• Development  Issues 

This  section  is  an  examination  of  the  nature  of  CT  and  its  characteristics  and  how  they  affect 
the  task  of  developing  and  offering  CT  services.  Section  2.1  presents  these  ideas  for  CT  in 
general,  and  STEP  CT  in  particular. 

In  any  standard-related  activity,  be  it  standard  development,  implementation,  or  CT,  there  are 
four  main  participants  that  are  primarily  involved.  These  are  the  standards  bodies  which 
develop  the  standards,  the  developers  who  will  build  standard-conformant  products,  the  users 
who  will  use  these  prtxlucts,  and,  last  but  not  least,  the  testing  community.  Of  these  four  main 
participants  each  has  its  own  concerns  and  needs  and  has  to  position  itself  strategically  in  order 
to  fill  these  needs.  Section  2.2  discusses  these  strategic  issues. 

Developmental  concerns  are  an  important  component  for  the  task  of  developing,  offering,  and 
arbitrating  CT.  Standards  bodies  and  user  groups  (who  are  interested  in  seeing  the  standard  get 
widespread  acceptance)  are  concerned  with  the  availability  of  high-quality  standards,  CT 
facilities,  and  competent  CT  staff.  Developers  want  to  ensure  CT  costs  are  reasonable,  and  CT 
is  conducted  fairly  and  without  prejudice.  Past  experience  has  shown  that  an  efficient  and  cost- 
effective  way  to  conduct  CT  for  networking  protocols  is  with  a set  of  tools  that  are  modular, 
configurable,  extendable,  and  automated.  For  STEP  this  is  particularly  significant.  The  right 
tools  that  can  be  configured  to  a given  ATS,  will  automate  the  development  of  executable  test 
cases  (ETCs)  and  save  on  resources  that  would  have  been  expended  in  developing  them 
manually.  Section  2.3  deals  with  these  issues  and  lists  brief  descriptions  of  paper-and-pencil 
tools,  software,  and  hardware  that  could  be  used  to  automate  the  CT  process. 


2.7  High-Level  Issues 

Conformance  testing  often  means  different  things  to  different  people.  Because  of  this  confusion 
there  is  a need  to  define  CT,  discuss  its  limitations,  and  describe  measures  of  success.  On 
some  occasions  CT  has  been  promoted  as  the  cure  for  incompatibility,  which  it  is  not.  On  the 
other  hand,  developers  may  see  it  as  an  extra  hurdle  in  the  path  to  commercialization.  The 
following  subsections  identify  some  important  issues  in  STEP  CT  and  provide  the  basis  for  the 
discussions  on  strategic,  developmental,  and  procedural  issues  that  follow. 


2,1.1  Scope  of  Conformance  Testing 

Successfully  completing  the  conformance  assessment  process  is  not  a guarantee  of 
interoperability.  CT  may  discover  errors  in  an  implementation;  it  cannot  however  demonstrate 
the  implementation  is  error  free.  For  a moment,  let  us  assume  that  all  conformance  faults  have 
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been  eliminated  for  a particular  implementation.  This  would  mean  that  an  implementation 
would  faithfully  follow  its  specified  behavior  and  that  behavior  required  by  the  standard.  But 
standards  have  their  own  inherent  (and  sometimes  unavoidable)  weaknesses.  Validation  and 
verification  of  STEP  APs  will  ho^fully  eliminate  the  most  serious  problems  of  this  nature  prior 
to  standardization,  and  thus  prior  to  CT.  Remaining  subtleties,  faults,  ambiguities,  or 
omissions  in  a standard  can  lead  to  two  conformant  implementations  failing  to  interoperate  in 
executing  a desired  function.  Thus,  demonstration  of  conformance  does  not  unequivocally 
translate  to  a guarantee  of  interoperability.  The  standards  development  process  must  balance 
two  diametrically  opposed  concepts:  (1)  over-specifying  requirements  to  guarantee 

interoperability;  or  (2)  under-specifying  requirements  to  provide  maximum  latitude  to 
implementors  to  optimize  performance  with  respect  to  cost. 

Conformance  testing  can  be  used  to  help  in  debugging  implementations.  ETSs  and  testing  tools 
developed  for  testing  laboratories  could  be  offer^  to  help  developers  in  such  debugging.  Also, 
CT  system  openness  (see  Section  2.2.2)  can  allow  developers  to  debug  implementations  at  their 
premises  without  using  accredited  CT  facilities. 


2,1,2  Why  Conformance  Testing  is  Useful 

The  four  main  participants  concerned  with  CT  have  different  views  of,  and  expectations  from, 
CT.  From  the  perspective  of  the  standards  bodies,  CT  is  key  to  facilitating  the  acceptance  and 
adoption  of  standards.  Without  CT  a standard  exists  only  on  paper,  and  vendors’  claims  of 
conformance  cannot  be  validated.  In  a sense,  the  conformance  test  is  the  standard.  Further, 
CT  can  provide  additional  validation  of  the  standard  itself.  Even  though  an  AP  was  tested 
before  being  established  as  a standard,  CT  may  yet  expose  parts  of  a standard  that  are  difficult 
to  implement,  contradictory  in  nature,  or  too  complex.  Thus  CT  can  provide  valuable  feedback 
to  standards  bodies  to  help  them  modify  standards  appropriately  and  thereby  facilitate 
acceptance  and  adoption  among  vendors  and  users. 

From  the  users’  point  of  view,  CT  gives  them  confidence  in  a product  in  much  the  same  way 
as  products  test^  and  marked  by  the  Underwriters  Laboratory— the  product  does  what  it  is 
intended  to  do.  In  the  long  run,  CT  aims  directly  at  reducing  costs  associated  with  product 
acquisition,  operation,  and  maintenance. 

In  order  to  build  conformance  products,  developers  must  invest  in  detailed  and  careful  design, 
test  thoroughly,  and  subject  the  product  to  CT.  Thus,  developers  often  see  CT  as  an 
impediment  in  their  ability  to  quickly  respond  to  market  needs,  and  as  a factor  that  raises 
product  prices.  However,  more  and  more  developers  agree  that  if  done  right,  an  early  start  in 
CT  makes  the  task  of  developing  standard-conformant  products  easier,  less  expensive  in  the 
long  run,  provides  a marketing  advantage,  and  gives  users  confidence  in  these  products. 

CT  services  and  tools  should  be  ready  when  STEP  is  accepted  as  an  international  standard. 
The  Initial  Graphics  Exchange  Specification  (IGES)  is  an  example  where  formal  conformance 
testing  requirements,  tools,  and  procedures  do  not  exist  today  for  a standard  against  which  there 
are  many  implementations.  Having  no  conformance  requirements  against  which  to  gauge 
developmental  requirements  has  proven  costly  to  the  vendor,  and  therefore  costly  to  the  user. 
In  addition,  users  are  often  reluctant  to  accept  vendor  claims  of  product  conformance  and 
therefore  marketing  of  the  product  is  delayed. 
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2.1.3  Conformance  Versus  Interoperability  Testing 

Traditionally,  for  standards  involving  the  interaction  of  different  companies’  products,  pairs  of 
vendors’  implementations  were  test^  to  see  if  they  interoperated.  This  process  is  loiown  as 
interoperability  testing.  To  correct  problems  with  the  two  systems’  interactions,  modifications 
were  then  made  to  either  product  (or  both)  based  on  what  was  convenient.  Thus,  a de  facto 
standard  evolved  as  a few  key  developers  iteratively  developed,  tried  to  interoperate,  went  back 
to  the  drawing  board  to  modify  their  implementation,  and  documented  the  assumptions  based 
on  their  interactions.  This  approach  was  acceptable  for  three  reasons:  there  were  relatively 
few  standards,  few  interested  developers,  and  no  better  method  was  known. 

Today,  standards  exist  or  are  under  active  development  in  many  areas  of  computing  and 
information  technology.  Also,  the  number  of  developers  and  users  has  increased.  But  most 
important,  the  benefits  of  systematic  development  and  standardization  of  CT  have  been 
recognized. 

CT  for  STEP  is  intended  to  determine  the  degree  to  which  a developer  has  correctly 
implemented  a standard  AP.  CT  determines  whether  or  not  an  implementation  operates 
correctly  both  in  normal  operation  and  under  error  conditions.  While  CT  evaluates  a single 
implementation  against  a reference  system,  interoperability  testing  uses  two  real  systems  to  see 
if  they  interoperate.  Since  candidate  systems  test  against  a single  reference  system  (or  exact 
copies  thereof,  CT  is  a parallel  process;  whereas  interoperability  testing  requires  interoperation 
between  all  systems,  thereby  having  a sequential  nature.  Doing  interoperability  testing  alone 
quickly  loses  its  attractiveness  as  the  number  of  systems  increases.  In  general,  the  cost  of 
interoperability  testing  grows  exponentially.  On  the  other  hand  CT,  while  starting  with  a higher 
fixed  cost  (because  of  the  need  to  build  the  reference  system),  grows  approximately  in  a linear 
fashion  with  N.  This  relationship  is  shown  in  Figure  1 . Interoperability  testing  is  not  without 
its  benefits,  however.  Often,  interoperability  testing  uncovers  problems  that  went  undetected 
during  CT. 


figure  1:  Confonance  Testing  versus  Interoperability  Testing 


5 


It  should  be  mentioned  that  neither  conformance  nor  interoperability  testing  is  transitive  in 
nature;  i.e.,  if  systems  A and  B pass  CT  with  reference  system  R,  there  is  no  guarantee  that 
they  will  interoperate.  Similarly,  if  A interoperates  with  B,  and  B interoperates  with  C,  there 
is  no  guarantee  that  A and  C will  interoperate.  However,  CT  puts  individual  systems  through 
the  same  transitions,  helping  to  take  care  of  nonconformant  problems  before  the  vendor 
participates  in  interoperability  testing.  Therefore,  doing  thorough  CT  before  attempting 
interoperability  testing  makes  sense. 

A iwsitive  aspect  of  CT  is  that  once  it  is  done  on  an  implementation,  it  need  not  be  repeated 
until  the  implementation  or  abstract  test  suite  is  modified.  The  intent  is  to  exercise  as  much 
of  the  logic  inherent  in  the  implementation  as  is  practical.  The  implementation’s  responses  to 
both  correct  and  incorrect  application  protocol  sequences  are  evaluated.  Due  to  the  complexity 
of  the  application  protocols,  it  may  not  be  realistic  to  try  to  be  exhaustive  during  CT  but  to  use 
a representative  sample  of  operational  sequences.  There  needs  to  be  a strong  agreement  among 
standards  bodies,  users,  and  developers  as  to  what  this  sample  for  a particular  application 
protocol  should  be.  These  operational  requirements  should  be  specified  in  the  standardized 
abstract  test  suites  (ATSs). 

An  early  start  in  STEP  CT  activity  will  likely  lead  to  economy  and  timely  realization  of  the 
benefits  of  product  data  standardization.  This  early  start  should  be  applied  to  writing 
standardized  CT  procedures,  developing  conformance  r^uirements  and  test  purposes,  and 
producing  supporting  ATSs  for  APs.  Interoperability  testing  will  have  to  be  done  later,  when 
independently  developed  systems  start  interacting.  With  strong  CT  activity  already  in  progress, 
interoperability  testing  will  be  a task  of  lesser  complexity  and  smaller  magnitude;  a lesson 
learned  from  the  IGES  user  community.  If  no  conformance  testing  is  available,  IGES  users  are 
forced  to  depend  solely  on  interoperability  testing  each  and  every  time  they  acquire  a new 
system.  With  no  frame  of  reference  for  determining  how  well  a product  conforms  to  a 
standard,  the  user  is  dependent  on  trial-and-error  to  obtain  a level  of  confidence. 


2.1,4  The  Cost  of  Conformance  Testing  Versus  the  Cost  of  Not  Doing 
Conformance  Testing 

There  are  a number  of  costs  associated  with  CT: 

• Building  the  organizational  infrastructure  for  doing  CT.  This  involves  acquiring  a 
site,  equipping  it,  and  maintaining  administrative  support  for  abstract  and  executable 
test  suite  development  and  later  CT. 

• Developing  CT  expertise.  This  involves  recruiting  appropriately  qualified  personnel 
and  training  them  in  the  related  CT  technology. 

• Building  CT  tools,  ATSs,  and  the  ETSs.  This  is  the  most  important  and  technology- 
intensive part.  This  component  will  take  up  the  largest  of  the  fixed  cost  and,  if  not 
controlled,  may  defeat  the  very  purpose  of  CT  by  becoming  very  expensive.  In 
the  context  of  STEP,  it  assumes  even  more  imiwrtance;  with  several  APs,  an 
economical  way  to  generate  high  quality  AP-specific  ATSs  and  ETSs  is  needed. 
The  way  to  achieve  this  is  to  build  tools  that  will  automate  the  process  of  building 
the  ATSs  and  ETSs  as  much  as  possible.  Since  work  is  already  underway  to 
validate  the  APs  prior  to  their  standardization,  tools  are  being  built  to  facilitate  this 
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process.  These  tools  should  be  examined  for  their  potential  use  when  building  the 
CT  system. 

• Maintaining  liaison  with  the  rest  of  the  world.  This  includes  participating  in 
standards  development  meetings  (e.g.  , ISO  TC184/SC4)  and  technology  development 
bodies  (e.g.  those  doing  AP  validation),  and  keeping  in  touch  widi  the  user  and 
vendor  communities.  The  CT  community  needs  to  be  actively  represented  in 
standards  bodies  and  implementation  agreement  bodies,  as  much  to  make  its  task 
simpler  as  to  provide  v^uable  input  to  the  standard  and  technology  development 
process.  This  participation  should  continue  not  just  when  the  ATS  and  ETS  are 
being  developed,  but  during  CT  development  and  enhancement.  Maintaining  contact 
with  the  users  and  vendors  is  needed  to  promote  CT,  its  value,  and  its  availability. 
Constant  contact  with  users  and  vendors  builds  confidence  in  the  CT  process. 
Also,  the  CT  community  can  be  a valuable  resource  to  users  and  vendors. 

• Maintaining  and  offering  the  CT  service.  The  tools,  ATSs,  and  ETSs  have  to  be 
maintained,  upgraded,  and  enhanced  from  time  to  time  to  reflect  the  revisions  and 
additions  to  the  standards.  The  CT  service  itself  involves  using  acceptable  testing 
facilities  and  maintaining  a CT-proficient  staff. 

• Transferring  CT  technology.  Here  we  assume  the  model  of  one  initial  CT 
development  center,  which  will  later  transfer  the  technology  to  multiple  testing 
laboratories  (see  Section  4.3).  In  the  case  of  STEP,  multiple  testing  sites  will  most 
likely  be  necessary  given  the  anticipated  large  number  of  users  and  vendors. 

The  items  identified  above  all  contribute  to  the  fixed-cost  component  (see  Figure  1)  of  CT.  The 
first  three  however,  are  the  major  contributors.  Careful  design  and  development  of  CT  tools 
are  an  important  means  of  reducing  and  controlling  fixed  costs.  Also  important  is  structuring 
the  APs  to  reduce  duplication  of  ATSs  and  the  need  to  retest  the  common  aspects  of  APs.  This 
is  why  active  representation  in  the  standards  bodies  is  required.  It  is  possible  to  come  up  with 
some  rules  of  diumb,  based  on  past  experience,  for  estimating  the  above  costs.  W^ile  a 
laudable  goal,  estimating  CT  costs  is  beyond  the  scope  of  this  report. 

The  cost  of  not  doing  CT  is  directly  associated  with  the  consequences  of  being  without  it. 
First,  the  earlier  approach  of  doing  ad  hoc  interoperability  testing  will  have  to  be  followed  in 
order  to  have  the  standard  implemented  and  used.  Alternatively,  there  is  the  distinct  possibility 
of  not  having  a standard  adopted  at  all.  For  instance,  the  lack  of  any  CT  for  IGES  has  been 
a contributing  factor  in  making  its  use  frustrating,  its  adoption  slow,  and  its  cost/benefit  ratio 
reduced.  Due  to  the  scope  of  STEP  and  the  number  of  APs  anticipated,  STEP  CT  will  be  a 
large  venture.  Conformance  testing  is  likely  to  be  a key  issue  in  the  success  of  STEP.  The 
long-term  costs  of  not  doing  CT  in  the  United  States  would  probably  retard  the  adoption  of 
STEP  in  the  United  States.  This  could  directly  hurt  the  U.S.  economy  and  have  a significant 
negative  impact  on  U.S.  international  trade  as  well.  The  European  Community  and  Japan  have 
made  a commitment  to  standards  in  general,  and  to  STEP  in  particular. 


2.1,5  Standards  for  Libraries  of  Data 

Conformance  testing  evaluates  implementations  for  conformance  to  standards.  In  the  case  of 
STEP,  CT  will  generally  use  databases  that  contain  descriptions  of  parts  and  components. 
These  data  repositories  or  libraries  must  themselves  be  standards-conformant.  For  example. 
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the  geometric  information  contained  in  a STEP  data  file  should  be  accessible  by  a STEP- 
conformant  CAD  system.  Apart  from  the  development  of  data  libraries  to  help  in  conducting 
CT,  hardware  vendors  are  likely  to  build  data  libraries  as  well.  STEP  data  files  must  be 
readable  by  the  implementation  under  CT.  In  addition  to  standards  for  STEP  data  files,  there 
should  be  standards  for  libraries  of  such  STEP  data  files. 


2.1.6  Conformance  Levels 

Past  experience  has  demonstrated  that  allowing  levels  of  partial  conformance  will  only  lead  to 
chaos  and  confusion  in  the  minds  of  users.  Partially  conformant  applications  may  be  less  likely 
to  interoperate,  thereby  undermining  the  most  fundamental  purpose  of  CT.  Moreover,  they 
encourage  misrepresentation  and  misuse.  Finally,  attempts  to  grant  partial  conformance  wiU 
render  the  task  of  developing  CT  systems  and  doing  CT  much  more  complex. 

Recommendation 

• Partial  conformance  must  be  treated  carefully;  full  conformance  must  be  the  goal. 

Generally,  products  would  either  be  conforming  or  nonconforming.  In  other  standards 
there  have  been  multiple  specification  classes  to  which  applications  may  choose  to 
conform.  In  the  STEP  standard,  current  strategy  seems  to  be  to  enforce  total  conformance 
to  a given  AP.  The  logic  behind  this  is  that  a well-defined,  stand-alone  part  of  an  AP 
should  become  a new  AP.  Provisional  nonconformance  could  be  considered  if  time  limits 
(e.g.,  one  year)  for  correcting  the  nonconformities  were  enforced.  Allowing  provisional 
nonconformance  permits  greater  competition,  results  in  more  products  in  the  market,  and 
lessens  the  "upfront  costs"  burden  to  the  implementor.  Ultimately,  within  an  AP,  full 
conformance  should  be  required. 


2.1.7  Dependencies  Between  Application  Protocols 

The  primary  objective  of  STEP  is  to  provide  for  meaningful  communication  of  product  data 
from  one  industrial  application  to  another.  In  STEP,  APs  will  be  interrelated.  The  scopes  of 
proposed  APs  should  be  reviewed  carefully  to  ensure  no  more  or  less  is  supplied  than  the 
justified  industrial  need  stated  on  the  AP  project  approval  summary  sheet.  This  raises  the  issue 
of  testing  a product  for  conformance  to  referenced  APs  to  the  extent  the  product’s  conformant 
behavior  may  be  affected  by  these  references.  The  question  of  how  far  to  test  and  when  to  stop 
is  not  directly  addressed  by  STEP.  In  TTI’s  opinion,  the  standards  development  activities  in 
STEP  have  not  progressed  far  enough  to  help  provide  answers  to  this  question. 

Recommendations 

• Further  study  is  needed  to  determine  how  APs  are  to  interoperate  and  what  the 
implications  on  CT  are. 

• The  ISO  TC184/SC4  working  groups  on  CT  and  AP  methodologies  need  to  address 
in  the  proposed  technical  report  for  AP.  Development  Guidelines,  the  issue  of  AP 
interoperability  and  how  it  relates  to  CT . 
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2.2  Strategic  Issues 


An  objective  of  the  DoD’s  CALS  (Computer-aided  Acquisition  and  Logistic  Support)  initiative 
is  to  encourage  the  adoption  and  use  of  STEP  in  DoD  procurements.  To  make  this  a reality, 
developers  and  users  will  have  to  have  confidence  in  STEP  and  work  to  build  skills  in  using 
it.  A key  objective  is  to  protect  the  interests  of  the  various  participants  involved  in  the  adoption 
of  STEP.  These  participants  are  the  developers,  the  users,  the  standards  bodies,  and  the  testing 
agents.  The  following  subsections  discuss  in  detail  the  various  issues  that  confront  these  four 
main  participants  in  relation  to  CT  in  general,  and  STEP  CT  in  particular. 


2,2,1  U,S,  Involvement  in  STEP  Conformance  Testing 

There  are  several  reasons  for  the  United  States  to  be  involved  in  STEP  CT.  Encouraging  CT 
activity  is  a way  to  build  indigenous  application  protocol  expertise.  This  will  prove  invaluable 
later  in  the  development  of  conformant  implementations  and  enable  U.S.  vendors  to  acquire  and 
maintain  a competitive  advantage  and  encourage  users  to  adopt  STEP-based  products. 

Since  CT  inevitably  raises  issues  about  the  standard  itself,  it  becomes  an  important  instrument 
in  the  evolution  of  the  standard.  The  United  States  needs  to  play  an  active  role  in  CT  and  the 
evolution  of  STEP. 

A significant  U.S.  effort  in  development  and  testing  activities  would  make  clear  the  U.S. 
government’s  intentions  and  reinforce  earlier  policy  statements  in  support  of  the  standard.  This, 
in  turn,  would  provide  the  impetus  needed  for  corporate  (both  user  and  developer)  buy-in.  The 
anticipation  of  future  business  needs  and  the  need  to  be  prepared  for  them  would  thus  trigger 
a development  of  STEP  expertise.  Another  important  benefit  of  indigenous  CT  activity  is  ^at 
the  organizations  involved  in  it  become  a resource  focused  on  making  STEP  a reality.  Vendors 
and  developers  can  use  this  resource  for  help  in  standards  implementation  and  sorting  out 
interoperability  and  acceptance  issues. 

Finally,  developing  early  CT  capability  for  an  AP  brings  with  it  the  opportunity  of  early 
understanding  of  the  standard.  For  STEP,  the  ability  of  U.S.  vendors  and  users  to  leverage  this 
opportunity  to  best  advantage  could  have  significant  economic  implications.  Conversely,  if 
left  to  others,  this  advantage  is  lost,  and  a dependence  on  non-U. S.  testing  bodies  would  result. 

U.S.  involvement  in  STEP  CT  will  benefit  the  entire  development  cycle  from  the  development 
of  early  conformant  implementations  to  the  creation  and  nurturing  of  a vigorous  and  competitive 
market  for  products.  Ultimately,  this  will  lead  to  faster  adoption  of  STEP  by  U.S.  industry 
through  the  rapid  appearance  of  more  and  better  STEP-conformant  products. 

Recommendation 

• It  is  important  that  a strong  participation  in  CT  be  part  of  U.S.  STEP  activities. 
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2,2.2  CT  System  Openness 


The  idea  that  CT  systems  should  be  open  has  considerable  merit.  Openness  of  the  CT  system 
means  CT  ATSs,  ETSs,  associated  test  tools,  and  associated  documentation  are  readily  available 
to  all  interested  parties.  Developers  and  users  could  use  the  CT  system,  voice  their  concerns 
about  problems  and  inconsistencies,  report  bugs,  and  thereby  provide  feedback  for 
improvement. 

Early  availability  of  the  CT  system  will  encourage  early  development  of  STEP-conformant 
products.  This  will  improve  product  quality  and  r^uce  developer  costs.  Familiarity  with  the 
CT  system  and  use  during  the  development  process  will  give  developers  a sense  of  confidence 
in  and  partnership  with  the  CT  process. 

Concern  has  been  voiced  that  making  the  CT  systems  completely  open  might  tempt  developers 
to  devote  more  of  their  energies  to  passing  the  CT  and  less  to  building  a better  product.  This 
concern  is  not  supported  by  experience  in  other  domains.  Complete  CT  systems  are  now 
available  for  many  networking  protocols,  and  commercial  organizations  are  building  their  own 
testing  laboratories  for  internal  testing  and  quality  assurance.  Competitive  pressures  force 
vendors  to  produce  good  products. 

An  easily  accessible  CT  system  will  expedite  disseminating  and  popularizing  the  standard.  It 
will  also  help  users  determine  their  own  requirements. 

Recommendations 

• Make  CT  systems  available  for  use  to  interested  developers  at  nominal  cost. 

• Create  a forum  for  interested  parties  to  contribute  toward  a better  CT  program. 
Inputs  for  improving  the  standards,  the  CT  system,  and  implementation  methods 
could  be  gathered  through  such  a forum. 


2,2.3  DoD  Security  Concerns 

Defense  organizations  and  contractors  will  develop  STEP-conformant  software  applications  as 
a result  of  the  CALS  initiative.  Often  such  software  will  work  in  a secure  environment  or 
access  secure  or  classified  resources  (like  databases).  Three  methods  performing  CT  on  such 
applications  exist:  (1)  installing  identical  systems  in  the  testing  laboratory  for  the  duration  of 
the  conformance  assessment  process;  (2)  "taking"  the  testing  laboratory  to  the  application;  or 
(3)  allowing  vendor  "self-testing"  as  declaration  of  conformity.  In  each  of  these  situations  the 
security  of  such  applications  could  be  compromised.  Performing  CT  on  sensitive  applications 
and  implementations,  given  their  nature  and  the  access  restrictions  they  carry,  is  a major 
concern. 

Recommendations 

• Two  alternative  recommendations  to  this  problem  are  suggested: 

DoD  develops  its  own  CT  laboratory  and  does  all  classified  testing  in  it. 


10 


DoD  secures  a security-cleared  contractor  who  acquires  CT  capability  and 
provides  the  service  to  DoD  in  a secure  environment. 

It  is  difficult  to  select  the  best  choice;  the  first  approach  may  be  suitable  if  there  is 
sufficient  software  to  be  tested  but  may  take  some  time  to  implement.  A contracting 
arrangement  might  prove  to  be  more  economical,  especially  if  the  volume  of  CT  does  not 
warrant  a dedicated,  in-house  lab. 


2,2.4  DoD  Adoption  Concerns 

The  Adoption  Issue;  DoD  is  probably  the  largest,  single  stakeholder  in  STEP.  STEP  is  also 
an  important  part  of  CALS.  DoD  wants  to  see  suppliers  adopt  STEP  so  that  consistent  means 
of  prcxluct  information  exchange  can  be  established.  Therefore,  a favorable  perception  of  STEP 
by  DoD  contractors  is  needed  to  ensure  adoption.  The  entire  CT  effort  is  aimed  at  achieving 
this  objective  by  expediting  the  adoption  process  and  making  it  happen  more  efficiently  and 
cost-effectively. 

Recommendation 

• DoD  should  set  realistic  adoption  milestones  and  actively  support  their  achievement. 

DoD,  as  a huge  buyer  of  information  technology  goods  and  services,  wields  enormous 
influence  with  its  suppliers.  DoD  could  use  this  influence  to  drive  vendor  agreement  to 
build  STEP  application  protocols  and  use  them  to  meet  DoD  procurement  requirements. 


2.2.5  Vendor  Community  Needs 

Fulfilling  the  needs  of  vendors  and  developers  would  go  a long  way  toward  encouraging 
widespread  adoption  of  standards  and  education  about  CT.  To  many,  CT  represents  a relatively 
new  approach  to  working  with  standards.  It  is  not  really  new.  Underwriters  Laboratory  has 
been  doing  this  kind  of  work  for  years  on  hardware  implementations.  The  Federal  Government 
has  been  doing  CT  of  computer  programming  language  software  implementations  for  years. 
Even  Consumers  Union  does  CT  on  a target^,  ad  hoc  basis.  It  is  important  to  promote  CT 
and  inform  the  broad  STEP  community  of  its  benefits.  This  would  help  vendors  understand  the 
need  for  conformance-tested  products  and  the  competitive  advantage  that  accrues  from  CT. 
This  promotional  task  has  to  be  a continuous  process. 

As  vendors  start  developing  STEP  conformant  products,  there  will  be  a need  for  identified 
personnel  that  wiU  answer  standcU’d-related  questions,  help  in  application  protocol 
implementation  and  CT  issues,  and  provide  a forum  for  discussion  and  problem  solving. 

Recommendations 

• A forum  to  discuss  CT  issues  should  be  created.  Standards  bodies  typically  provide 
such  a facility  in  some  form  or  another  (annual  or  semi-annual  conferences, 
publications,  newsletters,  bulletin  boards,  etc.),  but  an  additional  forum  for  CT  is 
necessary. 
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Outputs  from  such  a forum  would  include:  regularly  published  news  and  information, 
workshops,  education  for  vendors  and  developers,  and  promotion  of  CT.  This  forum 
would  provide  an  opportunity  to  disseminate  information  and  allow  developers  to  report 
problems,  request  information,  and  seek  related  assistance  to  perform  better  in-house 
testing. 

Seminars  and  workshops  to  do  training  on  CT  tools  and  methods  should  be  organized 
through  this  same  forum. 

• For  vendors  to  use  the  open  CT  systems  as  recommended  above,  training  to  use  the 
CT  tools  and  methods  is  necessary. 


2.2.6  Accreditation 

Here  we  assume  the  following  model:  Initially  there  would  be  one  team  for  developing  the  CT 
system.  Since  a lot  will  depend  on  this,  the  long-term  effects  of  choosing  a particular  team 
should  be  kept  in  mind.  Once  the  CT  system  has  been  developed  to  a level  where  CT  can 
begin,  multiple  testing  laboratories  would  be  set  up  and  accredit^. 

Recommendation 

• Since  one  of  NIST’s  primary  Junctions  is  to  promote  standards  for  government  use, 
it  should  be  considered  as  a candidate  to  accept  the  responsibility  of  directing  and 
performing  the  initial  CT  development. 

Therefore,  while  the  funding  agencies  and  NIST  should  work  together  to  come  up  with 
a consensus  approach  for  accrediting  testing  laboratories,  detailed  decisions  could  be  left 
to  NIST  as  an  impartial  agent. 

World-wide  development  of  STEP  CT  does  have  United  States  representation,  but  in  limited 
strength  relative  to  the  rest  of  the  world.  It  is  important  for  the  United  States  to  coordinate  its 
position  in  forums  such  as  ISO.  The  United  States  cannot  ignore  harmonization  activities  and 
therefore  must  organize  itself  into  a position  of  unity,  confidence,  and  strength  with  the  rest  of 
the  world.  Government  participation,  leadership,  and  teaming  with  industry,  as  is  common 
within  Europe  and  Japan,  would  go  a long  way  in  this  area. 


2.2. 7 Assigning  Conformance  Testing  Responsibilities  to  Testing  Laboratories 

Let  us  assume  staff  of  the  National  PDES  Testbed  (NPT),  working  with  others,  will  initially 
develop  the  CT  system.  They  would  then  distribute  the  CT  system  and  transfer  the  technology 
to  organizations  that  want  to  be  in  the  business  of  being  third-party  or  in-house  vendor  test 
centers,  once  accredited,  such  organizations  would  be  designated  as  testing  laboratories. 

For  widespread  adoption  of  a standard,  there  has  to  be  a commitment  shared  by  interested 
parties  to  make  the  standard  work.  A testing  laboratory  would  have  to  shoulder  its  share  of  this 
commitment.  The  most  fundamental  role  of  the  testing  laboratory  is  to  see  that  implementations 
work. 
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From  the  standard  development  point  of  view,  a testing  laboratory  should  also  play  a 
participatory  role  by  providing  fe^back  to  the  standard  development  process.  Such  input 
should  reflect  CT  issues  and  closely  address  the  problems  of  product  developers. 

Another  level  of  responsibility  of  testing  laboratories,  though  indirect,  may  be  to  ensure  that 
STEP,  as  a standard,  is  efficiently  and  economically  implemented.  Failure  to  satisfy  such 
efficiency  and  economy  criteria  often  impacts  how  the  standard  itself  is  perceived.  This  aspect 
is  beyond  the  scope  of  CT,  but  particularly  helpful  in  promoting  STEP’S  usefulness.  Efficiency 
and  economy  issues  of  implementations  are  often  dealt  with  in  workshops  between  testing 
laboratory  personnel  and  vendors. 

Recommendation 

• Ensure  applicant  testing  laboratories  are  both  committed  and  skilled  in  conformance 
testing  at^  the  technical  aspects  of  STEP. 


2.2.8  Abstract  Test  Suites  (ATSs) 

Leaving  unassigned  the  responsibility  of  developing  ATSs  would  result  in  different  organizations 
(developers,  testing  laboratories,  ad-hoc  industry  groups  or  consortia)  building  their  own  and 
claiming  conformance  to  them.  This  would  lead  to  inconsistencies  in  products  and  CT,  cause 
confusion,  and  ultimately  slow  down  the  adoption  process. 

Recommendations 

• Standards  bodies  should  be  responsible  for  the  ATSs. 

• ATSs  should  be  normative  parts  of  STEP. 

• The  ATS  should  be  a normative  requirement  for  a given  AP. 

The  actual  development  of  ATSs  could  be  done  by  anyone  (testing  laboratories,  consortia, 
etc.)  as  long  as  they  are  reviewed  by  the  standards  committees  and  accepted  as  part  of  the 
standards.  Developers  of  such  ATSs  should  be  committed  to  keeping  the  ATSs  in  the 
public  domain  and  internationally  approved  as  part  of  STEP. 


2.2.9  Executable  Test  Suites  (ETSs) 

It  is  possible  to  build  multiple  ETSs  from  the  same  ATS.  The  need  for  this  may  arise  due  to 
various  platform  and  implementation  choices  made  by  product  developers. 

Recommendation 

• Despite  the  non-unique  nature  of  an  ETS,  the  development  of  the  first  ETS  for  an 
ATS  should  be  done  by  those  building  the  ATS,  or  in  close  cooperation  with  them. 

This  would  help  to  identify  and  correct  any  problems  with  future  versions  of  the  standard 
ATS. 
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2,3  Development  Issues 


A number  of  issues  have  to  be  addressed  before  a strong  CT  infrastructure  is  built  and  brought 
into  operation.  The  main  resource  needs  of  CT  are  those  of  expertise,  tools,  a lead 
organization,  and  funds  to  support  such  activities.  Building  STEP  CT,  whether  in  the  United 
States  or  elsewhere,  will  require  people  with  the  appropriate  technical  expertise.  The  first 
major  task  of  this  CT  system  development  team  would  be  to  specify,  design,  build,  and  test  the 
CT  system  and  associated  documentation.  There  will  be  a need  for  an  organizational 
infrastructure  to  support  these  activities.  All  these  tasks  also  require  funding. 

There  needs  to  be  a technical  liaison  between  the  CT  system  development  team  and  ISO 
TC184/SC4.  Contact  has  to  be  maintained  through  active  participation  in  SC4  Working 
Groups.  Contact  also  has  to  be  maintained  with  organizations  and  industry  groups  developing 
ATSs.  To  keep  abreast  with  vendor  activities  and  issues  of  interpretation,  CT  system 
development  staff  should  attend  implementors’  agreement  meetings,  trade  shows,  and  symposia. 

Another  set  of  tasks  includes  continually  assessing  and  controlling  the  quality  of  the  CT  systems 
being  built  by  the  CT  development  team.  Only  the  initial  stages  of  CT  system  and  tool 
development  will  be  handled  by  the  CT  development  team  as  a single  entity.  Once  the  early 
versions  of  the  transferable  CT  system  have  been  developed,  many  organizations  would  have 
the  opportunity  to  become  testing  laboratories.  These  organizations  should  be  accredited  and 
a standard  procedure  should  be  established  to  do  this. 


2,3,1  Abstract  Test  Suite  (ATS)  Development 

It  has  been  discussed  earlier  that  standards  bodies  should  include  developing  the  ATSs  as  part 
of  the  standards-making  process.  ATSs  could  be  developed  by  any  independent  organization, 
but  responsibility  for  their  ratification  and  acceptance  belongs  to  ISO  TC184/SC4. 

Recommendations 

• When  the  CT  development  activity  begins,  some  initial  start-up  funding  may  be 
needed  to  pay  for  the  development  of  ATSs  for  those  APs  released  without  an  ATS. 

• Once  the  CT  activity  stabilizes,  the  cost  of  developing  ATSs  for  new  (and  changing) 
APs  should  be  borne  by  the  users  of  the  APs.  For  example,  ATSs  for  APs  supporting 
the  automotive  industry  should  be  paid  for  by  the  auto  industry. 

One  approach  to  ATS  development  is  that  one  center  would  develop  the  CT  systems  and  tools 
for  a small,  self-contained  piece  of  STEP  (like  a group  of  related  APs).  This  would  give  the 
primary  sponsor  of  the  activity  opportunity  for  input  and  assure  interested  parties  the  task  is  not 
getting  bogged  down  in  the  complexities  of  very  large  system  development.  Starting  small 
would  make  it  possible  to  monitor  the  development  activity,  keep  quality  high,  and  keep  costs 
down.  As  the  task  progresses,  the  scope  could  increase.  This  approach  towards  incremental 
development  fits  well  with  all  the  strategic  issues  discussed  in  Section  2.2. 
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2.3.2  Funding  Conformance  Testing 


Conformance  testing  start-up  costs  have  generally  been  borne  by  interested  industry  groups  or 
consortia  and  applications  developers.  Of  course,  any  costs  faced  by  developers  eventually 
filter  down  to  the  user  in  the  form  of  higher  prices.  This  effect  would  be  more  significant  if 
developers  are  expected  to  pay  for  the  start-up  as  well,  since  the  development  of  the  CT  activity 
would  require  a significant  initial  investment.  Ultimately,  the  benefits  of  CT  to  users  of  STEP 
implementations  have  to  be  seen  as  greater  than  the  cost.  If  users  fund  the  start-up  of  a CT 
service,  the  developer  may  incur  a lower  fee  for  such  a service.  Low  fees  will  attract 
developiers  and  help  get  the  CT  activity  off  the  ground.  Once  stabilized,  the  rise  in  revenue 
from  CT  should  make  it  possible  to  reduce  the  subsidy.  In  the  long  run  CT  should  be  a 
vendor-supported  activity.  Therefore,  if  CT  costs  are  reasonable,  it  encourages  more  vendors 
to  build  affordable  implementations  while  still  undergoing  CT. 

There  are  other  issues  to  consider: 

• Does  the  complexity  of  the  application  protocol  and  magnitude  of  the  ETS 
affect  fees? 

• Do  all  vendors  pay  the  same  or  do  smaller  companies  pay  less  than  larger 
ones. 

Recommendation 

• When  conformance  testing  first  begins,  vendors  should  not  be  expected  to  pay  more 
than  nominal  sums  to  have  their  products  tested. 


2.3.3  Legacy  Issues 

Retesting  may  be  needed  when  developers  claim  conformance  with  newer  versions  of  standards, 
or  modify  their  products  to  the  existing  standards.  Thus,  a changing  standard  or  product  drives 
the  need  for  retesting.  There  are  two  issues  to  consider:  availability  and  cost.  If  the  CT 
systems  are  easily  available  (Section  2.2.2),  vendors  could  do  some  sort  of  strictly  controlled 
provisional  conformance  testing  in-house,  if  their  original  base  system  was  already  deemed 
conforming.  This  would  drive  costs  down.  Also,  the  CT  assessment  process  costs  should  be 
reasonable  to  encourage  vendors  to  get  CT  done  for  upgraded  products  early.  Moreover,  if  CT 
is  carefully  modularized,  retesting  may  only  involve  CT  of  those  portions  that  have  undergone 
changes. 

Recommendation 

• As  mentioned  earlier,  if  CT  is  done  right  (open,  modular,  reasonably  priced), 
retesting  should  not  be  an  insurmountable  problem.  Also,  as  APs  stabilize,  retesting 
would  cease  to  be  an  important  issue. 
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2,3,4  Tools 


One  point  this  report  has  emphasized  is  the  need  for  automating  the  conformance  assessment 
process  and  the  ETS  development  process.  The  way  to  achieve  this  is  through  software  tools. 
CT  tools  reduce  dependence  on  the  expertise  of  the  CT  system  operator  and  also  reduce  the 
chances  of  CT  system  operator  error.  CT  development  tools  help  build  new  CT  system 
components,  thus  reducing  costs. 

The  following  is  a list  of  the  tools  needed  for  conformance  testing.  They  can  be  classified  into 
three  types:  CT  System  Tools,  Client  Support  Tools,  and  CT  Service  Tools.  Some  of  the 
tools  listed  below  will  become  a part  of  any  CT  system  for  virtually  any  application  protocol. 
Others  are  less  common,  but  valuable,  nonetheless. 


2, 3, 4,1  CT  System  Tools 

Test  Engines  - These  are  the  generally  invariant  core  drivers  of  test  execution  that  probe  and 
detect  the  behavior  of  the  System  Under  Test  (SUT).  They  perform  common  activities  required 
by  the  abstract  and  executable  test  cases  (ATCs  and  ETCs). 

Executable  Test  Cases  - These  are  the  data  sets  or  instructions  that  when  coupled  with  a test 
engine,  control  a test  engine  into  stimulating  particular  sets  of  behaviors  from  an 
Implementation  Under  Test  (lUT), 

Test  Session  Controllers  - These  tools  sequence  and  orchestrate  the  execution  of  other  CT 
tools,  with  the  objective  to  see  the  procedural  requirements  of  conformance  test  sessions  are 
properly  carried  out,  or  if  not,  duly  noted. 

Loggers  - These  tools  record  the  most  basic  activities  of  CT  system  execution  for  consumption 
by  other  tools,  or  for  an  occasional  detailed  audit  of  test  execution  events. 

Monitors  - These  are  the  real-time  sensors  and  formatters  which  observe  and  report  on  SUT 
behavior. 

Analyzers  - These  are  tools  which  substitute  for  both  tedious  clerical  editing  and  insightful 
expert  interpretation. 

Report  Generators  - These  tools  collect  the  important  outputs  of  the  other  CT  system  tools  into 
a report  format  suitable  for  external  distribution. 

Preparation  Tools  - These  tools  collect  information  about  an  SUT,  and  record  it  for  use  by 
other  CT  system  components. 

Data  Management  Tools  - These  tools  enable  organized  archival  storage  and  examination  of 
test  data. 

Debuggers  - These  tools  are  a required  by-product  of  CT  system  development  in  that  they  can 
be  us^  to  evaluate  and  possibly  correct  CT  system  operation. 
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Build  Tools  - These  tcx)ls  extract  appropriate  information  about  the  tests  to  be  performed,  and 
configure  and  activate  the  CT  system  accordingly. 

Clean-Up  Tools  - These  tools  de-activate  and  de-configure  the  CT  system,  along  with  safely 
archiving  captured  test  results. 

Training  Tools  - These  tools  can  be  used  to  educate  the  testing  laboratory  staff  on  the  use  of 
the  CT  system. 


2,3 A. 2 Client  Support  Tools 

CT  System  Interfaces  - These  are  the  necessary  hooks  that  a client  must  provide  to  the  CT 
system  in  order  for  the  SUT  to  be  examined. 

Test  Support  Tools  - These  may  be  required  of  a client,  executable  in  the  SUT,  so  that  the  SUT 
is  testable. 

Test  Responders  - These  are  tools  and  functions  sometimes  required  by  vendors  to  test  their 
implementations.  These  could  be  sample  code  that  is  portable  or  a detailed  specification. 

Test  Procedures  - These  are  established  to  help  clients  get  their  products  tested  easily.  These 
procedures  should  be  precise,  simple,  and  easy  to  carry  out. 

Report  Generators  - These  are  tools  to  format  the  CT  results  for  easy  perusal  by  vendors  and 
users. 

Training  Tools  - These  tools  can  simplify  the  client’s  efforts  in  the  CT  operation  activities  by 
providing  key  information  about  such  aspects  as  the  CT  process  and  requirements  of  an  SUT. 


2. 3, 4. 3 Testing  Laboratory  Service  Tools 

Test  Engine  - As  described  above,  it  is  the  automated  apparatus  which  can  stimulate  and 
observe  an  SUT’s  behavior  in  a controlled  manner. 

System  Diagnostics  - These  tools  can  be  engaged  to  validate  proper  operation  of  the  CT 
system,  the  client  system,  or  the  CT  environment. 

Remote  Testing  - These  tools  can  be  used  to  execute  tests  over  telecommunication  facilities  if 
required  by  special  client  needs. 

Remote  Control  - These  tools  further  enhance  remote  CT  by  providing  clients  with  the 
capability  to  control  the  CT  environment  for  either  preconformance  testing  preparation  or 
debugging. 

Scheduling  - These  tools  are  the  classic  software  packages  which  are  quite  valuable  in 
moderate-to-high  activity  operations. 

Logistics  Coordination  - These  tools  help  in  facilitating  coordinated  interactions  between  CT 
service  parties. 


17 


Accounting  - These  tools  assist  in  tracking  the  costs  of  both  client  and  testing  laboratory 
operations. 

Forms  Management  - These  tools  assist  in  the  generation,  preparation,  and  organization  of 
blank  and  completed  reports,  forms,  and  data  sets. 

Report  Production  - These  tools  help  with  the  collection,  organization,  and  preparation  of 
reports  for  "official"  results  reporting,  CT  plan  generation,  and  other  external  document 
publication. 

Recommendation 

• The  specific  set  of  tools  needed  for  STEP  CT  needs  to  be  determined  as  part  of  the 
CT  system  development. 

A full  set  of  CT  tools  is  probably  too  much  to  develop  as  part  of  the  initial  CT  work. 
As  an  initial  set  of  specific  tools  is  develof^,  they  will  help  in  further  CT  and  product 
development.  The  earlier  tools  become  available,  the  quicker  CT  of  standard-conformant 
products  will  occur.  The  lists  presented  above  are  an  initial  suggestion  of  tools  likely  to 
be  useful. 
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3.  Evaluation  Procedures  for  STEP  Conformance  Testing 


The  success  of  STEP-based  application  systems  will  depend  on  the  representation  of  product 
data  in  a form  completely  independent  of  any  particular  computer-aided  software  system.  This 
basic  requirement  brings  with  it  the  need  for  file  exchange  capabilities,  sharing  of  product 
databases,  and  device-independent  archiving  and  de-archiving  capabilities. 

To  have  confidence  in  the  verdict  of  a CT  system,  the  system  itself  needs  to  be  evaluated  as 
do  the  ATSs  for  the  various  application  protocols  that  will  be  used  to  generate  the  ETS  run  on 
the  CT  system.  Thus,  evaluation  applies  suitably  defined  criteria  to  candidate  CT  systems  and 
offers  authoritative  approval  or  disapproval.  In  this  section  we  focus  on  the  evaluation  of  CT 
systems,  ATSs,  and  ETSs  for  CT  and  offer  some  recommendations  for  evaluation  procedures. 

One  important  caveat  needs  to  be  raised  in  regard  to  CT  system  approval.  The  evaluation 
process  must  not  fall  into  a gatekeeper  ment^ty.  The  processes  need  to  be  designed  to 
encourage  CT  and  the  operation  of  testing  laboratories.  If  the  attitude  becomes  one  of  keeping 
testing  laboratories  out  rather  than  helping  them  become  operational,  then  the  CT  system  wifi 
be  inadequate,  if  not  counterproductive.  Likewise,  the  testing  laboratories  will  need  to  take  the 
approach  of  helping  their  customers  create  high-quality,  standards-conformant  products. 


3,1  CT  System  Evaluation 

Criteria  are  needed  for  the  process  of  evaluation  of  CT  systems.  The  following  subsections 
introduce  some  important  criteria  and  related  recommendations. 


3.1,1  CT  System  Environment 

The  hardware  and  software  environments  in  which  the  CT  system  exists  will  have  to  be 
evaluated.  Since  multiple  CT  environments  could  be  built  to  meet  the  criteria  for  acceptance, 
any  such  CT  system  must  not  give  commercial  advantage  to  any  particular  candidate  product 
during  CT. 

Recommendations 

• The  CT  environment  specifications  should  be  easily  available  to  all  interested  parties 
who  would  have  the  opportunity  to  voice  their  concerns,  if  any,  and  request 
reasonable  modifications  in  the  early  stages  of  CT  system  development. 

• The  acceptance  of  a CT  environment  should  not  favor  some  product  vendors  over 
others. 

• The  CT  environment  must  be  an  "open " platform  to  and  from  which  the  porting  of 
CT-related  software  will  be  easily  accomplished. 

• The  CT  environment  should  not  impose  architectural  requirements  on  SUTsfor  their 
successful  CT;  i.e.,  the  cause  for  failure  of  a test  or  the  inability  to  perform  it 
should  not  be  architectural  incompatibility  between  the  CT  system  and  the  SUT. 
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3,1,2  CT  System  Availability 


Product  vendors  need  easy  access  to  CT  systems  to  be  used  as  development  and  debugging 
tools  throughout  the  process  of  product  development.  Thus  CT  systems  should  be  reasonably 
priced. 

Recommendation 

• The  cost  of  acquiring  (biding,  leasing,  or  sharing)  a CT  system  should  not  be  such 
that  it  would  prohibit  widespread  use. 

Since  the  cost  of  purchasing  some  CT  systems  may  preclude  their  in-house  use  by  some 
vendors,  arrangements  to  make  them  available  at  r^uced  costs  or  for  cost  sharing  should 
be  put  in  place. 


3,1,3  CT  System  Characteristics 

CT  systems  need  to  possess  some  basic  characteristics  which  assure  their  integrity,  exhibit  their 
independence  of  the  CT  environment,  and  provide  evidence  of  their  usability: 

• Repeatability  of  CT  results 

• Reporting  and  easy  comparability  of  CT  results 

• Auditability  of  CT  results 

• Usability  (ease  of  installation  and  use)  of  the  CT  system 


3,1,3, 1 Repeatability  of  Conformance  Testing  Results 

To  assure  a high  level  of  confidence  in  the  results  of  any  conformzmce  assessment  process  and 
the  credibility  of  the  testing  laboratory,  CT  results  must  be  repeatable.  A combination  of  the 
same  ETS,  CT  system,  and  the  lUT  must  produce  identical  results  over  multiple  runs. 

Recommendations 

• A candidate  CT  system  should  be  subjected  to  a calibration  process  to  evaluate  its 
repeatability.  This  evaluation  should  include  the  CT  system’s  responses  to  both 
error-free  and  erroneous  implememations  of  STEP  products. 

To  maintain  a high  degree  of  confidence  in  the  performance  of  this  calibration,  the 
implementation(s)  to  be  used  should  be  selected  from  a known,  fairly  large  list.  The 
results  of  this  exercise  will  form  the  basis  for  the  judgement  of  whether  repeatable  results 
can  be  expected. 

• Because  repeatability  of  test  results  may  be  affected  by  CT  system  operator  error 
and  result  generation  error,  a high  degree  of  automation  is  recommended. 
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3,13,2  Reporting  and  Easy  Comparability  of  Conformance  Testing  Results 

A CT  system  must  report  information  that  will  allow  the  production  of  a Conformance  Test 
Report  (CTR)  for  each  implementation  tested  against  each  relevant  AP.  The  CTR  shall  specify 
for  every  test  run  one  of  three  outcomes:  pass,  fail,  or  inconclusive.  No  other  verdicts  are 
allowed.  The  CT  system  should  thus  allow  logging  of  each  test  outcome  during  the  testing 
process. 

Moreover,  the  CTR  generated  should  be  independent  of  the  CT  environment.  Thus  for  the 
same  SUT,  the  CTR  generated  at  different  test  sites,  whether  in-house  or  in  a testing 
laboratory,  should  be  comparable. 

Recommendations 

• The  CT  system  should  possess  the  capability  of  using  its  logs  to  automatically 
generate  a CTR. 

• Formats  and  procedures  as  defined  in  ISO  WD  10303-32^  for  generating  CTRs 
should  be  adhered  to. 

• Procedures  in  ISO  WD  10303-34*  for  executing  test  cases  should  be  followed. 

These  procedures  should  be  built  with  flexibility  in  mind  and  should  consider  situations 
in  which  repetitions  of  specific  test  case  executions  may  be  warranted. 


3, 1,3, 3 Auditability  of  Conformance  Testing  Results 

A CT  process  usually  includes  an  arbitration  subprocess.  To  effectively  arbitrate  a dispute 
between  product  vendor  and  testing  laboratory,  detailed  information  on  the  inputs  and  outputs 
for  the  execution  of  a test  suite  is  required. 

Recommendation 

• The  ability  to  document  the  actual  procedure  of  a test  campaign  and  log  its  details 
should  be  built  into  the  CT  system.  These  logs  should  then  be  used  as  the  basis  of 
the  arbitration  process. 


3, 1,3, 4 Usability  of  the  Conformance  Testing  System 

The  CT  system  builder  should  not  assume  that  the  CT  system  operator  will  be 
STEP-knowledgeable  or  conversant  with  CT  technology.  It  should  be  possible  to  quickly  train 
a lay  computer  user  to  be  a CT  system  operator.  Also,  the  purposes  for  building  and  using  a 
CT  system  is  defeated  if  many  parts  of  the  conformance  assessment  process  are  left  non- 


’ISO  WD  10303-32,  "The  Requirements  on  Testing  Laboratories  and  Clients." 
^ISO  WD  10303-34,  "Abstract  Test  Methods." 
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automated.  The  CT  system  should  be  straightforward  to  install,  configure,  and  operate.  CTRs 
should  be  clear,  concise,  and  should  contain  all  required  information. 

Recommendations 

• TTie  CT  system  should  be  easy  to  use  and  learn.  At  the  time  of  building  the  CT 
system,  minimal  assumptions  ^out  the  CT  system  operator’s  skills  should  be  made. 
The  CT  system  operator  command  set  should  be  compact  and  relatively  simple  to 
master  and  use.  Information,  error  messages,  and  prompts  should  be  clear  and 
easily  interpretable. 

• The  CT  system  should  incorporate  tools  to  automate,  as  far  as  possible,  the  entire 
CT  process  thereby  minimizing  the  room  for  human  error. 

• Comprehensive  documentation  should  be  available  for  installation,  configuration  and 
use  of  the  CT  system.  The  CIR  generators  should  follow  the  format  requirements 
of  ISO  WD  10303-32  for  CTRs. 


3,1.4  Equivalence  of  CT  Systems 

The  issue  of  CT-system  equivalence  assumes  importance  when  multiple  CT  systems  for  the 
same  set  of  application  protocols  have  to  be  evaluated.  The  primary  criterion  is  to  show 
comparability  of  conformance  test  results  between  CT  systems.  The  first  step  towards 
achieving  this  is  to  use  a common  ATS. 

While  it  is  not  possible  to  formally  prove  the  equivalence  of  two  CT  systems,  multiple  ETSs 
with  identical  SUTs  provide  a reasonable  level  of  confidence  that  future  lUT  CT  results  would 
be  comparable.  At  least  one  of  the  SUTs  has  to  be  non-conforming  to  ensure  the  system  can 
detect  problems  properly. 

Demonstrating  the  equivalence  of  CT  systems  with  a high  degree  of  confidence  is  technically 
challenging  and  demands  rigorous  examination  of  conformance  test  suite  logs  and  results. 
Identic^  circumstances  have  to  be  maintained  on  each  CT  system  throughout  the  evaluation 
process  and  the  maximum  possible  amount  of  data  logged  for  post-completion  evaluation. 

Recommendations 

• Where  a reference  implemeiuation  capable  of  generating  known  errors  in  a 
controlled  manner  exists,  it  should  be  used  in  the  equivalence  evaluation  process. 

• The  ETCs  should  be  carejully  planned,  run  in  identical  sequences,  and  the  results 
and  logs  subjected  to  meticulous  examination. 

• If  a CT  system  is  modified  (in  any  manner)  by  its  developer,  it  should  again  be 
subjected  to  the  same  process. 
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3,2  Executable  Test  Suite  Generation 


It  is  a matter  of  general  agreement  that  the  standards  committee  that  develops  a particular  AP 
will  also  oversee  the  development  of  the  ATS  for  that  AP.  Once  officially  accepted,  the  ATS 
will  then  be  used  by  testing  laboratories  to  develop  ETSs.  These  tasks  will  be  performed  in 
accordance  with  the  ISO  TC184/SC4  10303-30  class  of  standards  which  specify  different 
abstract  test  methods,  development  of  ATSs,  and  the  policies  and  proc^ures  for  the 
development  of  ETSs  for  STEP.  That  an  ETS  correctly  implements  its  parent  ATS  is  a matter 
important  enough  to  warrant  attention  by  the  Certification  Body. 


3,2,1  Policy  Issues 

The  need  for  openness  in  STEP  CT  systems  requires  the  availability  of  detailed  information  on 
the  CT  system,  ATSs,  and  ETS  tools.  Also,  the  need  for  strictly  adhering  to  standards  requires 
documents  and  forms  be  filled  out  before,  during,  and  after  the  test  campaign.  It  is  the 
mandate  of  the  CT  system  builder  to  provide  sufficient  assurance  of  quality  and  coverage  in 
areas  which  the  standard  specifies  as  required. 

Recommendations 

• The  CT  system  builder  must  provide  all  documentation  for  the  implementation  of  the 
AP  to  be  tested.  Such  documentation  should  provide  all  information  necessary  to 
prepare  the  SUT  for  conformance  testing  and  include  a description  of  the 
conformance  assessment  process,  a definition  of  the  CT  system  configuration,  and 
a description  of  any  special  requirements  placed  on  the  SUT  by  the  CT  system. 

• The  ETS  should  test  lUT  response  to  both  valid  and  invalid  behavior  by  the  CT 
system.  Invalid  behavior  should  include  illegal  parameters  and  invalid  parameter 
values. 

• The  conformance  testing  procedures  used  by  the  testing  laboratory  should  be  made 
public  to  help  ensure  that  the  conformance  assessment  process  is  valid. 


3,2,2  Abstract  and  Executable  Test  Suite  Coverage 

One  major  point  of  evaluation  of  the  CT  system  must  be  an  analysis  of  how  well  the  CT  system 
covers  the  ATS  (and  therefore  the  ETS)  of  the  AP  to  which  conformance  is  being  tested.  Since 
the  ATS  development  process  will  have  public  input,  most  concerns  about  coverage  will  be 
ironed  out  for  the  ATS  during  the  standardization  process.  However,  deficiencies  could  be 
identified  with  the  use  of  the  ETS  and  improved  coverage  sought. 

Recommendation 

• An  AP  coverage  method  should  be  used  to  establish  how  well  an  ATS  covers  its 
associated  AP. 

Empirical  methods  of  measuring  AP  coverage  are  available.  One  such  method  is 
specified  in  the  m technical  report  "Test  Coverage  Analysis  and  Measurement  (TCAM): 
A Practical  Approach  to  Determining  Coverage"  (document  m TR-87-14.1). 
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3.3  CT  System  Builder 


The  organization  charged  with  building  a CT  system  has  to  satisfy  certain  important  criteria; 
a commitment  to  STEP  and  independence  from  all  individual  participants  in  the  STEP 
community.  Moreover,  the  CT  system  builder  must  also  make  specific  commitment  to  maintain 
the  CT  system  and  undertake  adequate  quality  assurance  measures. 


3.3.1  Commitment 

The  CT  system  builder  must  indicate  its  long-term  commitment  to  STEP  and  demonstrate  its 
ability  to  support  the  CT  system.  Modifications  to  a CT  system  are  an  ongoing  process  for  any 
number  of  reasons.  The  CT  system  configuration  may  be  impacted  by  new  versions  of 
hardware  and  software;  although  the  CT  system  will  have  undergone  evaluation  prior  to 
release,  continual  debugging  is  often  necessary  as  more  implementations  undergo  CT;  or 
changes  to  an  AP  or  its  ATS  may  warrant  modification  to  system  code.  If  the  CT  system 
builder  chooses  not  to  continually  maintain  the  CT  system,  a breakdown  in  the  CT  service  may 
occur. 

Recommendations 

• The  CT  system  builder  should  he  committed  to  keeping  the  CT  system  up-to-date  with 
the  base  standards  and  APs  as  they  change. 

• The  CT  system  builder  must  maintain  adequate  liaison  with  several  organizations  to 
keep  abreast  with  ongoing  issues. 

These  organizations  include:  standards  committees,  p^r  conformance  testing  organizations 
abroad,  and  user/vendor  forums  and  consortia.  Maintaining  such  contact  enables  the  CT 
system  builder  to  anticipate  future  trends  and  needs,  and  act  promptly  to  respond  to  them. 


3.3.2  Nonalignment 

The  product  developers  who  will  use  the  conformance  testing  service  must  be  assured  that  the 
CT  system  builder  is  an  independent  body  with  no  direct  commercial  affiliations.  This  is 
important  to  maintain  impartiality  of  the  CT  system  and  also  to  encourage  the  developers  to 
utilize  the  conformance  testing  service  to  improve  their  products. 

Recommendation 

• An  independent  organization,  such  as  NIST,  should  be  charged  with  the 
responsibility  of  providing  the  STEP  CT  system  and  associated  tools.  In  performing 
this  task,  the  body  should  act  to  remain  nonaligned  and  make  acquisition,  design, 
and  implementation  decisions  in  keeping  with  this  mandate. 
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3.3.3  Maintenance y Revision,  and  Q^ality  Assurance 

As  in  any  other  complex  project  involving  hardware  and  software,  the  CT  system  builder  has 
to  maintain  adequate  controls  on  the  revision  of  the  CT  system  and  on  error  tracking  and 
fixing.  Since  users  and  vendors  would  be  able  to  buy  or  lease  CT  systems  for  in-house  use, 
they  should  also  have  access  to  points  where  they  can  report  problems. 

Once  a CT  system  is  accepted  for  use,  there  has  to  be  a mechanism  for  ensuring  its  continued 
quality.  The  organization  that  charter^  the  CT  system  builder  should  independently  assess  the 
quality  of  the  CT  system. 

The  CT  system  builder  should  be  able  to  respond  to  errata  and  other  changes  to  the  standards 
(APs).  Changes  in  the  CT  system  should  be  made  in  an  orderly  manner  and  all  known  users 
should  be  informed  of  new  CT  system  releases. 

Recommendations 

• The  revision  control  policies  and  schemes  for  the  CT  system  builder  should  be 
reviewed  by  user  and  vendor  groups  to  determine  if  they  achieve  adequate  quality 
control. 

• The  CT  system  builder  should  have  procedures  in  place  to  receive  problem  reports 
and  act  on  them  in  a timely  manner. 

If  the  problem  is  with  the  ATS  or  AP,  the  ISO  TC184/SC4  community  and  all  known 
users  should  be  notified  immediately  and  work  begun  to  fix  it.  If  the  CT  system  is  the 
problem,  work  to  rectify  it  should  be  started  and  users  notified.  All  problems  and  fixes 
should  be  logged. 

• TTw  CT  system  builder  should  have  procedures  in  place  for  tracking  errata  and  other 
changes  in  the  APs  and  determining  the  impact  of  these  changes  on  the  CT  system 
and  ATSs. 

If  changes  are  required,  the  nature  of  the  work,  expected  problems,  and  projected  time 
of  completion  should  be  made  known  to  all  users  of  the  CT  system. 

• CT  system  releases  should  be  accompanied  by  comprehensive  documentation 
including:  system  fixes  since  last  release,  AP  changes  affecting  the  new  release,  and 
remaining  known  problems. 

• The  organization  that  initiates  and  funds  developing  the  CT  system  must  have  in 
place  procedures  by  which  problems  and  complaints  could  be  independently  accepted 
and  progress  monitored.  Procedures  to  resolve  such  problems  have  to  be  set  up 
and  implemented. 
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The  CT  system  builder  should  consider,  and  CT  services  and  procedures  reflect,  the 
quality  assurance  requirements  identifled  in  the  ISO  9000  series^  or  equivalent. 


3,4  The  CT  System  Evaluation  Process 

The  CT  system  builder  may  desire  to  maintain  a committee  (comprised  of  representatives  from 
CT  participating  organizations)  with  the  mandate  of  evaluating  all  relevant  aspects  of  the  CT 
system  and  the  CT  system  builder  (see  previous  sections).  Only  after  satisfactory  completion 
of  these  procedures  would  the  CT  system  builder  claim  recognition  of  its  CT  system.  The 
following  subsections  make  broad  recommendations  on  CT  system  qualification.  If  CT  systems 
are  to  be  developed  and  put  in  place,  the  burden  of  evaluation  must  not  be  too  heavy. 


3.4,1  Evaluation  of  the  CT  System 

The  CT  system  builder  would  seek  acceptance  of  its  CT  system  through  the  committee 
mentioned  in  Section  3.4.  The  committee  would  evaluate  the  CT  system.  This  evaluation 
should  be  based  on  criteria  that  have  been  discussed  in  the  sections  above  (repeatability, 
comparability  of  test  results,  etc.).  All  evaluation  done  during  this  process  would  use  ETS 
tools  already  approved  by  the  committee. 

Throughout  the  evaluation  process,  the  committee  must  work  with  the  CT  system  builder  to 
resolve  any  problems  encountered  without  compromising  the  quality  and  integrity  of  the  CT 
system  itself.  Upon  acceptance  of  the  CT  system,  complete  and  detailed  documentation  would 
be  released  to  show  that  the  CT  system  meets  the  criteria  for  recognition. 


3.4.2  Recognition 

The  granting  of  recognition  would  be  done  for  a specific  CT  system  and  a specific  set  of  APs, 
the  versions  of  each  being  specified.  Recognition  would  be  valid  until  specifically  withdrawn 
by  the  committee  which  constantly  monitors  CT  activity.  Withdrawal  of  recognition  would 
occur  if  it  were  determined  that  the  CT  system  builder  failed  to  show  commitment  to  maintain 
the  CT  system  and/or  adequately  perform  the  usual  tasks  of  bug  fixing,  upgrading,  and  user 
support. 


*ISO  9000,  Quality  management  and  quality  assurance  standards  - guidelines  for  the  selection  and  use. 
ISO  9001,  C^lity  systems  - model  for  quality  assurance  in  design/development,  production,  installation 
and  servicing. 

ISO  9002,  Quality  system  - model  for  quality  assurance  in  production  and  installation. 

ISO  9003,  Quality  systems  - model  for  quality  assurance  in  final  inspection  and  test. 

ISO  9004,  (^lity  management  and  quality  system  elements  - guidelines. 

[The  above  international  standards  are  available  through  the  ISO  Secretariat:  1,  rue  de  Varembe,  Case 
postale  56,  CH-1211  Geneva  20,  Switzerland.] 
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4.  Milestones 


The  previous  sections  in  this  report  emphasized  that  development  of  STEP  application  protocols 
and  associated  implementations  r^uire  the  availability  of  conformance  testing  capabilities  to 
reap  the  benefits  of  standardization.  It  is  essential  to  have  in  place  a conformance  testing 
capability  within  the  United  States  as  early  as  possible.  The  Unit^  States  should  thus  have  an 
aggressive  STEP  CT  development  program  of  its  own,  not  relying  on  other  countries.  The 
benefits  are  many:  bringing  competitiveness  to  the  industry,  nurturing  the  growth  of  expertise 
at  home,  and  strengthening  the  position  of  U.S.  members  in  intemationtd  standards  bodies. 

The  first  step  in  developing  STEP  CT  capabilities  in  the  United  States  is  to  come  up  with  a 
plan  and  a realistic  schedule  for  CT  system  development  and  deployment.  Such  a plan  should 
be  in  line  with  the  needs  of  major  STEP  users  and  should  take  into  account  the  current  state 
of  the  standards  development  efforts.  This  section  enumerates  a set  of  milestones  for 
developing  and  deploying  a STEP  CT  system  and  starting  a CT  service.  These  milestones 
should  be  coordinated  with  the  NIST  Development  Plan  for  STEP  CT  Services.** 

There  are  three  major  task  groups: 

• CT  System  Development 

• CT  System  Validation 

• Technology  Transfer  and  Testing  Laboratory  Accreditation 


4.1  CT  System  Development 

This  subsection  details  the  milestones  for  the  task  of  CT  system  development. 


4.1.1  Generating  a Requirements  and  Development  Plan 

This  milestone  will  be  the  culmination  of  a requirements  analysis  task  that  will  focus  on  three 
important  issues  affecting  design  and  development  of  the  CT  system.  The  milestone  will  be  the 
generation  of  a high-level  requirements  document,  a development  plan,  and  an 
inter-organization  advisory  team.  Together  these  will  provide  important  input  to  the  system 
design  process  and  focus  on  building  the  capabilities  to  test  conformance  to  the  selected  APs. 

First,  general  system  requirements  need  to  be  identified  to  feed  into  later  tasks.  This  task  will 
solicit  input  from  interested  parties  (users,  vendors,  standards  bodies)  on  what  is  needed  in  a 
CT  system.  Early  involvement  in  ^e  CT  process  will  serve  to  unify  the  participants’  view 
and  offer  a forum  for  discussion.  Also,  users’  concerns  will  be  taken  into  account  from  the 
start.  This  task  will  be  done  by  requesting  information  from  all  concerned,  and  by  hosting  one 
or  more  workshops  to  discuss  and  evaluate  issues  and  select  options. 

Second,  the  CT  system  developer  should  invite  voluntary  participation  from  interested 
organizations.  These  organizations  may  want  to  seek  assurance  that  work  is  progressing  on 


‘Kemmerer,  Sharon,  National  PDES  Testbed  Development  Plan,  STEP  Conformance  Testing  Service. 
NISTIR  4641,  August  1991. 
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schedule  and  in  alignment  with  the  needs  of  the  user  and  vendor  communities,  activities  of  the 
standards  bodies,  and  international  STEP  activities. 

Third,  APs  have  to  be  chosen  for  the  CT  system.  This  choice  will  depend  on  the  priorities  of 
the  funding  agency  in  conjunction  with  prospective  users  of  the  technology.  As  far  as  possible, 
APs  that  have  advanced  to  or  are  close  to  becoming  draft  international  standards  (DIS)  should 
be  selected. 


4.1,2  Accept  Abstract  Test  Cases 

It  has  been  recommended  that  standards  bodies/committees  developing  the  APs  bear  the 
responsibility  of  generating  ATSs.  This  task  will  examine  the  ATSs  thus  generated. 
Ambiguities  that  could  affect  ATS  interpretation  during  the  development  of  ETSs  should  be 
brought  to  the  attention  of  ISO  TC184/SC4  and  resolved  at  this  stage.  This  examination  may 
also  analyze  coverage  and  make  recommendations  to  the  appropriate  committee  for  changes  to 
improve  coverage  or  reduce  redundancy. 


4.1.3  Define  or  Accept  Abstract  Test  Methods 

This  task  will  accept  abstract  test  methods  for  CT  of  STEP-based  implementations.  The  output 
will  be  a document  describing,  in  general  terms,  the  abstract  test  methods  that  will  be  used  for 
the  selected  APs. 

ISO  WD  10303-34  of  STEP  is  concerned  with  defining  and  describing  abstract  test  methods 
which  take  into  account  the  control  and  observation  points  of  STEP  implementations.  Since 
this  CT  system  development  task  will  be  the  first  for  STEP,  it  may  well  be  possible  that  for  the 
APs  selected,  the  abstract  test  methods  in  STEP  Part  34  may  need  to  be  modified  or  redefined. 


4.1.4  Generate  a Detailed  CT  System  Requirements  Specification 

This  task  will  develop  a requirements  specification  for  the  CT  system.  The  previous  four  tasks 
will  have  generated  implicit  and  explicit  requirements  for  the  CT  system. 

This  task  will  describe  the  needed  capabilities  of  the  CT  system.  Moreover,  the  requirements 
specification  should  be  flexible  enough  for  modification.  As  the  first  CT  effort  for  STEP,  it 
is  almost  certain  that  changes  will  be  required  as  work  on  STEP  progresses  and  new 
requirements  emerge. 


4.1.5  Develop  CT  System  Architecture  and  Design 

This  task  will  develop  a detailed  architectural  design  and  a detailed  internal  design.  The 
architectural  design  will  decompose  the  CT  system  into  system  modules,  interfaces,  and 
interconnections  and  provide  detailed  descriptions  of  these  items.  It  will  also  specify  the 
hardware  and  software  configuration  used  to  build  and  operate  the  CT  system. 
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The  detailed  internal  design  will  select  and/or  define  algorithms  and  data  structures  that  will  be 
used  to  build  the  CT  system  components  according  to  the  requirements  specification.  This  task 
will  also  consider  and  identify  existing  tools,  utilities,  and  services  to  efficiently  implement 
these  objects. 

The  plan  to  implement  the  CT  system  will  be  refined,  taking  into  account  evolving 
requirements  of  STEP  CT  as  STEP  becomes  standardized  and  priorities  change. 


4.1.6  Develop/Procure  Platform  and  Development  Tools 

The  output  of  this  task  will  be  a functional  hardware/software  platform  on  which  development 
can  begin  and  unit  testing  can  be  done.  As  the  development  of  the  CT  system  progresses, 
more  equipment  and  tools  may  have  to  be  added  to  the  platform  to  assess  the  capability  and 
functionality  of  the  modules  developed.  A detailed  list  of  the  kinds  of  tools  usually  needed  for 
CT  can  be  found  in  Section  2.3.4. 


4.1.7  Develop  Abstract  Test  Suites 

For  each  AP  selected,  an  ATS  is  necessary.  The  output  of  this  task  will  be  ATSs  for  the 
selected  APs  with  complete  and  comprehensive  documentation. 

An  ATS  will  have  to  be  accepted  and  supported  internationally  and  be  consistent  with 
worldwide  STEP  efforts.  Thus,  a procedure  to  process  existing  ATSs  through  ISO  must  be  put 
in  place. 

Some  of  the  ATSs  may  have  to  be  developed.  These  should  go  through  the  same  acceptance 
procedure  referred  to  above.  As  emphasized  before,  all  accepted  ATSs  should  be  well 
documented  and  internationally  standardized. 


4.1.8  Construct  and  Integrate  CT  System 

This  task  will  use  the  detailed  CT  system  architecture  and  design  as  input  to  build  the  CT 
system.  The  output  of  this  task  will  be  an  alpha  version  of  the  CT  system  accompanied  by 
comprehensive  documentation. 

This  is  a significant  development  component  of  the  CT  effort  and  will  incorporate  an  ETC 
generator  for  the  selected  APs;  compilers  and  parsers  for  the  STEP  description  language,  ISO 
CD  10303-11,  EXPRESS;  and  comprehensive  documentation  and  a capabilities  list  using  the 
requirements  specifications  as  a baseline.  As  far  as  possible,  the  components  of  the  CT  system 
will  be  unit-tested  as  they  are  built.  However,  substantial  system  testing  will  need  to  be  done 
separately  as  described  in  the  following  section. 
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4,2  CT  System  Validation 


After  completion  of  the  CT  system,  the  system  will  have  to  be  thoroughly  tested.  An 
undesirable  concept  is  to  build  a test  bed  that  is  bigger  than  the  CT  system  being  evaluated. 
Such  a process  could  spiral  out  of  control  and  make  the  whole  CT  concept  collapse  upon  itself. 


4,2.1  Develop  CT  System  Validation  Plan 

Developing  a validation  plan  for  the  CT  system  will  be  necessary.  In  the  absence  of 
(certifiably)  conformant  product  implementations  that  could  be  used  to  "reverse-validate"  the 
CT  system  and  also  to  avoid  any  controversy  in  the  STEP  community,  a limited,  independent 
assessment  is  appropriate.  The  output  of  this  task  will  be  a validation  process  and  an 
implementation  plan  to  validate  the  first  CT  system. 


4,2.2  Validate  CT  System 

This  task  will  use  an  independent  test  bed  to  validate  the  CT  system.  The  validation  and 
"fixing"  of  the  CT  system  will  be  done  iteratively;  problems  discovered  will  be  fixed  and  the 
CT  system  retested  starting  from  a reasonable  baseline.  The  output  of  this  task  will  be  a fully 
validated  CT  system  ready  to  be  used  to  perform  CT  on  product  implementations  based  on  the 
selected  APs. 


4.3  Technology  Transfer  and  Testing  Laboratory  Accreditation 

The  STEP  CT  system  developed  in  the  above  tasks  will  represent  a major  step  toward  building 
a credible  capability  to  perform  CT  on  STEP-based  products.  This  CT  capability  will  be  akin 
to  developing  new  technology,  may  bring  with  it  new  testing  approaches,  and  will  represent  a 
significant  advancement  in  the  area  of  standardized  product  data  representation. 

To  disseminate  this  new  technology  and  leverage  its  utility  many  times  over,  it  has  to  be 
transferred  to  testing  laboratories  that  would  then  independently  perform  CT.  To  begin  to 
assess  the  quality  and  credibility  of  these  testing  laboratories,  one  may  choose  to  ensure  the 
testing  laboratories  chosen  for  the  task  have  a reputation  for  quality  work  in  areas  related  to 
STEP.  It  is  also  necessary  that  the  CT  technology  be  transferred  to  the  testing  laboratories  in 
the  right  manner  and  that  the  testing  laboratories  possess  a certain  level  of  expertise  in  the 
technology  area  in  which  they  are  offering  CT  services.  Thus,  it  is  necessary  to  have  in  place 
detailed  procedures  for  technology  transfer  and  testing  laboratory  accreditation.  This  task  could 
use  the  GOSIP  or  POSIX  accreditation  methods  as  models. 


4.3.1  Establish  a Conformance  Testing  Service 

The  first  task  is  to  start  a CT  service  at  the  site  where  the  CT  system  was  developed.  This 
would  assess  the  new  procedures  as  well  as  further  evaluate  the  CT  system  before  release  to 
other  sites.  Offering  this  service  requires  detailed  procedures  to  perform  CT  along  with  forms 
for  CT  reports  and  logs,  means  of  arbitration,  and  fee  structures.  This  task  develops  a 
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framework  for  offering  CT  services  including:  technical,  commercial,  administrative,  and  legal 
guidelines.  It  also  includes  advertising  and  initiating  STEP  CT  services. 


4.3.2  Develop  Technology  Transfer  Plan 

As  discussed  above,  technology  transfer  will  be  an  important  task  in  making  STEP  CT  services 
available.  This  task  develops  a plan  to  transfer  the  CT  technology  to  candidate  testing 
laboratories. 


4.3.3  Develop  Testing  Laboratory  Accreditation  Procedure 

This  task  develops  procedures  to: 

• Select  testing  laboratories  for  transfer  of  the  CT  technology  based  on  their  technical, 
commercial,  and  administrative  capabilities. 

• Assess  and  evaluate  the  testing  laboratories  on  the  basis  of  their  exf^rtise  in  STEP, 
quality  and  experience  of  personnel,  adequate  equipment  and  quality  control,  and 
adherence  to  CT  procedures. 

Too  much  bureaucracy  in  this  process  will  lead  to  high  costs,  a gatekeeper  mentality,  and 
ultimately,  failure  in  creating  a useful  CT  system. 


4.3.4  Invite  and  Approve  Candidate  Laboratories 

This  task  advertises  for  testing  laboratory  applicants.  It  will  evaluate  these  applicants  on  the 
basis  of  the  criteria  described  in  task  4.3.3  above.  The  output  of  this  task  is  a set  of  testing 
laboratories  that  have  passed  the  selection  criteria. 


4.3.5  Start  Up  Conformance  Testing  Service 

This  task  transfers  the  CT  technology  to  the  selected  testing  laboratories.  The  output  of  this 
task  is  a register  of  testing  laboratories  with  a list  of  the  APs  for  which  they  offer  CT  services. 
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5.  Summary  and  Conclusion 


Developing  STEP  conformance  testing  capability  will  be  an  important  task  in  the  near  future. 
This  report  discussed  the  important  issues  and  established  a baseline  for  high-level  needs  and 
requirements,  CT  system  evaluation  procedures,  and  milestones  for  building  and  offering  a 
STEP  CT  system  in  the  United  States. 

The  issues  discussed  in  this  report  have  emphasized  the  following: 

• Appropriate  conformance  testing  is  an  economical  proposition  in  the  long  run  cuid 
should  be  pursued  actively  in  order  to  encourage  adoption  of  the  standard. 

• Although  developing  and  establishing  facilities  for  CT  is  expensive,  the  long-term 
cost  of  not  doing  CT  is  even  more  expensive. 

• Conformance  testing  may  be  needed  for  implementations  that  utilize  databases  of 
product  data  as  well  as  exchange  implementations. 

• For  the  United  States  to  rely  on  other  countries  to  undertake  CT  would  lessen  the 
capability  of  American  companies  to  compete  in  the  international  marketplace. 

• The  needs  of  the  STEP  product  vendor  community  must  be  addressed,  with  open 
CT  systems  being  an  important  contribution. 

• CT  software  tools  are  a very  important  component  of  CT  systems.  They  automate 
CT  tasks  and  help  develop  new  ATSs  and  supporting  ETSs. 

• Evaluation  of  CT  systems  and  services  is  an  important  issue.  Evaluation  must  ensure 
that  CT  is  accurate,  honest,  and  fair  while  not  being  an  undue  burden  on  the  testing 
laboratories. 

• The  CT  system  builder  must  be  committed,  nonaligned,  and  provide  quality 
assurance. 


The  discussions  in  this  report  led  to  the  following  conclusions: 

• The  United  States  should  initiate  the  building  of  a STEP  CT  system. 

• The  STEP  CT  system  that  is  built  should  be  available  for  use  by  all  interested 
parties  at  a nominal  cost.  Easy  availability  will  help  produce  better  products, 
improve  the  level  of  vendors’  expertise,  and  build  user  and  vendor  confidence. 

• The  CT  system  that  is  built  should  be  highly  automated  and  also  include  utilities  and 
software  tools  to  automate,  to  a large  extent,  the  tasks  of  interpreting  CT  results, 
developing  ETSs,  and  generating  conformance  test  reports. 

• The  ISO  TC184/SC4  should  be  responsible  for  developing  and/or  ratifying  ATSs 
for  application  protocols. 


33 


• The  initial  CT  service  should  start  small  based  on  a limited  set  of  APs.  The  CT 
capability  should  expand  as  needed. 

• There  should  be  a process  to  address  the  needs  of  vendors  and  users  before  and 
during  the  design  and  development  of  the  CT  system.  The  main  goal  is  to  serve  the 
users. 
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6.  Glossary 


abstract  test  case  (ATC):  One  or  more  files,  encapsulating  the  test  purpose,  which  provide 
the  basis  from  which  parameterized  executable  test  cases  are  derived. 

abstract  test  method:  The  description  of  how  an  lUT  is  to  be  tested,  given  at  the  appropriate 
level  of  abstraction  to  make  the  description  independent  of  any  particular  implementation  of 
testing  tools  or  procedures,  but  with  enough  detail  to  enable  executable  test  cases  to  be 
generated  from  this  test  method. 

Gaboratory)  accreditation:  The  formalized  initial  and  continuing  process  of  ensuring  a testing 
laboratory  is  competent  to  carry  out  specific  (types  of)  tests. 

NOTE  - The  term  "laboratory  accreditation"  covers  the  recognition  of  both  the  technical 
competence  and  the  impartiality  of  a testing  laboratory.  Accreditation  is  normally 
awarded  following  successful  laboratory  assessment  and  is  followed  by  appropriate 
surveillance. 

accreditation  body:  A body  that  conducts  and  administers  a laboratory  accreditation  scheme 
and  grants  accreditation. 

NOTE  - An  accreditation  body  may  wish  to  delegate  fully  or  partially  the  assessment  of 
a testing  laboratory  to  another  competent  body  (assessment  agency).  While  it  is 
recogni^  that  this  may  be  a practical  solution  to  extending  recognition  of  testing 
laboratories,  it  is  essential  that  such  assessment  be  equivalent  to  that  applied  by  the 
accreditation  body  and  that  the  accreditation  body  t^e  full  responsibility  for  such 
extended  accreditation. 

certificate  of  conformity,  certificate  of  conformance:  A document  issued  under  the  rules  of 
a certification  system  indicating  that  adequate  confidence  is  provided  that  an  lUT  is  in 
conformity  with  a specific  stand^d  or  technical  specification  as  determined  through  use  of  a 
specified  abstract  test  method. 

certiHcation  body:  An  impartial  body  possessing  the  necessary  competence  and  reliability  to 
operate  a certification  system,  and  in  which  the  interests  of  all  parties  concerned  with  the 
function  of  the  system  are  represented. 

client  (of  a testing  laboratory):  The  organization  that  submits  an  implementation  for 
conformance  testing. 

comparability  (of  results):  Characteristic  of  conformance  assessment  processes,  such  that 
execution  on  the  same  SUT,  in  different  testing  laboratories,  leads  to  the  same  overall 
summary. 

conformance  assessment  process:  The  complete  process  of  accomplishing  all  conformance 
testing  activities  necessary  to  determine  the  conformance  of  an  implementation  to  an  application 
protocol. 

conformance  test  report:  A document  written  at  the  end  of  the  conformance  assessment 
process,  which  provides  both  summary  and  detailed  information.  The  first  part  gives  the 
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overall  summary  of  the  conformance  of  the  lUT  to  the  standard  for  which  conformance  testing 
was  carried  out;  the  second  gives  the  details  of  the  testing  carried  out  for  a particular  standard. 

conformance  testing:  The  testing  of  a candidate  product  for  the  existence  of  specific 
characteristics  required  by  a standard;  testing  the  extent  to  which  an  lUT  is  a conformant 
implementation,  (source:  ISO  WD  10303-1,  "Overview  and  Fundamental  Principles,"  August 
3,  1990.) 

executable  test  case  (ETC):  A realization  of  an  abstract  test  case.  (The  form  of  the 
realization  is  still  under  development.) 

GOSIP:  Government  Open  Systems  Interconnection  Profile. 

Implementation  Under  Test  (lUT):  That  part  of  a product  which  is  to  be  studied  under 
testing,  which  should  be  an  implementation  of  one  or  more  characteristics  of  the  standard(s). 

interoperability  testing:  Related  to  acceptance  testing,  but  specifically  applied  to  the 
examination  of  the  information  exchange  between  two  specific  lUTs  and  the  ability  of  each  lUT 
to  use  such  information. 

NOTE  - This  does  not  form  part  of  conformance  testing. 
lUT:  See  Implementation  Under  Test. 

POSIX:  Portable  Operating  System  Interface  for  Computer  Environments. 

repeatability  (of  results):  Characteristic  of  a test  case,  such  that  repeated  executions  on  the 
same  SUT  under  the  same  conditions  lead  to  the  same  test  verdict,  and  by  extension  a 
characteristic  of  a test  suite. 

SUT:  See  System  Under  Test. 

System  Under  Test  (SUT):  The  computer  hardware,  software,  and  communication  network 
required  to  support  the  lUT. 

test  campaign:  The  process  of  executing  the  (parameterized)  executable  test  suite  for  a 
particular  lUT. 

test  report:  See  conformance  test  report. 

test  suite:  A complete  set  of  test  cases,  possibly  combined  into  nested  test  groups,  that  is 
necessary  to  perform  conformance  testing  for  a standard  or  group  of  standards. 

Note:  This  term  is  deprecated  unless  prefixed  by  abstract  or  executable. 

testing  laboratory:  An  organization  that  carries  out  the  conformance  assessment  process.  This 
can  be  a third  party,  a user  organization,  or  an  identifiable  part  of  the  supplier  organization. 
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